---
id: sampling-tracetest-spans
title: Sampling Tracetest Spans
description: When running Tracetest in production, your test spans could not be sampled by your probabilistic sampler. Here's how to avoid sampling out Tracetest test spans.
keywords:
  - tracetest
  - trace-based testing
  - observability
  - distributed tracing
  - testing
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---

Suppose you are considering Tracetest to run tests against a high-volume environment, such as production. In that case, you probably will hit some drawbacks, such as having
your test spans not sampled by your probabilistic sampler. There are a couple of things that you can do to avoid those problems:

## Add a Separate Pipeline for Tracetest in your OpenTelemetry Collector

Your existing OpenTelemetry Collector already receives traces from your applications and sends them to your datastore and you have a set of processors configured to ensure the quality of the traces your datastore is receiving. It probably looks like this one:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 5.0

  batch:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/jaeger:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]
```

We don't want to change this pipeline for your testing to work, so let's create a new pipeline for collecting only traces generated by Tracetest tests:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 5.0

  # If this configuration fails on your collector, make sure to update it to a newer version.
  # This is the recommended way of filtering spans based on the `trace_state`. It's faster and less
  # resource intensive than using a `tail_sampling` approach.
  filter/tracetest:
    error_mode: ignore
    traces:
      span:
        - 'trace_state["tracetest"] != "true"'

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/jaeger:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]

  pipelines:
    traces/tracetest:
      receivers: [otlp]
      processors: [filter/tracetest, batch]
      exporters: [otlp/jaeger]
```

### Tail Sampling Approach
Before December 2023, we were suggesting people to use tail sampling to filter the traces generated by Tracetest.
However, the new `filter` capabilities are better for performance than tail sampling as it
requires less memory to decide if a trace should be sampled or not. If you are not concerned with memory usage and still want to use the
tail sampling approach, this is the processor you can use:

```yaml
processors:
  tail_sampling:
    decision_wait: 5s
    policies:
      - name: tracetest-spans
        type: trace_state
        trace_state: { key: tracetest, values: ["true"] }
```

With this configuration, you will still get 5% of all your traces, but you will also ensure that all your test traces are collected and sent to
Jaeger.

## Consider Trace-related Headers in your Ingress Endpoint

When exposing something to the internet, it's common to ignore all unknown headers from an HTTP request. This usually includes the `traceparent` and `tracestate` headers, which are the two headers defined in the [W3C specification](https://www.w3.org/TR/trace-context/#relationship-between-the-headers).

However, supporting those two fields can be beneficial. For starters, you can start testing your application in production using Tracetest. If your product APIs are used directly by your customers, you can also share the same Trace-ID as them and use this to correlate a faulty operation on their side with your trace.
